Papers with teacher PLMs
Knowledge Inheritance for Pre-trained Language Models (2022.naacl-main)
Copied to clipboard
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou
| Challenge: | Existing large-scale pre-trained language models are mainly trained from scratch individually, ignoring that many well-taught PLMs are available. |
| Approach: | They propose a pre-training framework called knowledge inheritance and propose auxiliary supervision to efficiently learn larger PLMs. |
| Outcome: | The proposed framework can be used to train large-scale language models with huge parameters and a large dataset can be adapted to domain adaptation and knowledge transfer. |
One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have huge model sizes and computational complexity, making it difficult to deploy them to low-latency and high-concurrence online systems. |
| Approach: | They propose a multi-teacher knowledge distillation framework for pre-trained language model compression that can train high-quality student model from multiple teacher PLMs. |
| Outcome: | The proposed framework can train high-quality student model from multiple teacher PLMs with shared pooling and prediction layers to align output space for better collaborative teaching. |